You can access the course materials quickly from

https://ayoubbagheri.nl/applied_tm/

Some guidelines

1- Please keep your microphone off

2- If you have a question, raise your hand or type your question in the chat

3- You may always interrupt me

4- We will introduce frequent question breaks

• Text data is everywhere!
• A lot of world’s data is in unstructured text format
• The course teaches
  • text mining techniques
  • using Python
  • on a variety of applications
  • in many domains.

• Can you guess what are the challenges Garry, Larry, and Harry encounter in doing their job, when working with text data?

  • Go to www.menti.com and use the code 7338 2184

• Huge amount of data

• High dimensional but sparse

  • all possible word and phrase types in the language!!

• Noisy data

  • Examples: Abbreviations, spelling errors, short text

• Complex relationships between words

  • “Hema merges with Intertoys”

  • “Intertoys is bought by Hema”

• Different things can mean more or less the same (“data science” vs. “statistics”)
• Context dependency (“You have very nice shoes”);
• Same words with different meanings (“to sanction”, “bank”);
• Lexical ambiguity (“we saw her duck”)
• Irony, sarcasm (“That’s just what I needed today!”, “Great!”, “Well, what a surprise.”)
• Figurative language (“He has a heart of stone”)
• Negation (“not good” vs. “good”), spelling variations, jargon, abbreviations
• All the above are different over languages, 99% of work is on English!

• We won’t solve linguistics …
• In spite of the problems, text mining can be quite effective!

• Which can be a part of Text Mining definition?
  • The discovery by computer of new, previously unknown information from textual data
  • Automatically extracting information from text
  • Text mining is about looking for patterns in text
  • Text mining describes a set of techniques that model and structure the information content of textual sources

(You can choose multiple answers)

Go to www.menti.com and use the code 7338 2184

• “the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources” Hearst (1999)

• Text mining is about looking for patterns in text, in a similar way that data mining can be loosely described as looking for patterns in data.

• Text mining describes a set of linguistic, statistical, and machine learning techniques that model and structure the information content of textual sources. (Wikipedia)

https://dh2017.adho.org/abstracts/079/079.pdf

https://asreview.nl/

• Text classification
• Text clustering
• Sentiment analysis
• Feature selection
• Topic modelling
• Word embedding
• Deep learning models
• Responsible text mining
• Text summarization

• Represent by a string?

  • No semantic meaning

• Represent by a list of sentences?

  • Sentence is just like a short document (recursive definition)

• Represent by a vector?

  • A vector is an ordered finite list of numbers.

• A vector space is a collection of vectors

• Represent documents by concept vectors

  • Each concept defines one dimension

  • k concepts define a high-dimensional space

  • Element of vector corresponds to concept weight

• Distance between the vectors in this concept space

  • Relationship among documents

• The process of converting text into numbers is called Vectorization

• Terms are generic features that can be extracted from text

• Typically, terms are single words, keywords, n-grams, or phrases

• Documents are represented as vectors of terms

• Each dimension (concept) corresponds to a separate term

\[d = (w_1, ..., w_n)\]

• All documents are projected into this concept space

• Split text into words and sentences

• N-grams: a contiguous sequence of N tokens from a given piece of text

  • E.g., ‘Text mining is to identify useful information.’

  • Bigrams: ‘text_mining’, ‘mining_is’, ‘is_to’, ‘to_identify’, ‘identify_useful’, ‘useful_information’, ‘information_.’

• Pros: capture local dependency and order

• Cons: a purely statistical view, increase the vocabulary size \(O(V^N)\)

Typical steps:

• Stemming (“running”\(\to\)“run”) or Lemmatization (“were”\(\to\)“is”)
• Lowercasing (“And”\(\to\)“and”)
• Stopword removal (“evning morning is third day.”)
• Punctuation removal (“evning morning is third day”)
• Number removal (“day 3”\(\to\)“day”)
• Spell correction (“evning”\(\to\)“evening”)
• Tokenization (“evening”, “morning”, “is”, “third”, “day”)

Not all of these are appropriate at all times!

• Unifies variations in the text data:
  • e.g., ’walking’, ‘walks’, ‘walked’ \(\to\) walk
• Inflectional stemming:
  • Remove plurals
  • Normalize verb tenses
  • Remove other affixes
• Stemming to root:
  • Reduce word to most basic element
  • More aggressive than inflictional
  • e.g., ‘denormalization’ \(\to\) norm;
  • e.g., ‘Apply’, ‘applications’, ‘reapplied’ \(\to\) apply

1. Tokenization:

D1: ‘Text’, ‘mining’, ‘is’, ‘to’, ‘identify’, ‘useful’, ‘information’, ‘.’

2. Stemming/normalization:

D1: ‘text’, ‘mine’, ‘is’, ‘to’, ‘identify’, ‘use’, ‘inform’, ‘.’

3. N-gram construction:

D1: ‘text-mine’, ‘mine-is’, ‘is-to’, ‘to-identify’, ‘identify-use’, ‘use-inform’, ‘inform-.’

4. Stopword/controlled vocabulary filtering::
D1: ‘text-mine’, ‘to-identify’, ‘identify-use’, ‘use-inform’

After tokenization and pre-processing, we have three options:

• Bag of Words

• Topics

• Word Embeddings

• With Bag of Words (BOW), we refer to a Vector Space Model where:

  • Terms: words (more generally we may use n-grams, etc.)

  • Weights: number of occurrences of the terms in the document

• Term as the basis for vector space

  • Doc1: Text mining is to identify useful information.

  • Doc2: Useful information is mined from text.

  • Doc3: Apple is delicious.

• Binary

  • with 1 indicating that a term occurred in the document, and 0 indicating that it did not

• Idea: a term is more important if it occurs more frequently in a document

• TF Formulas

  • Let \(t(c,d)\) be the frequency count of term \(t\) in doc \(d\)

  • Raw TF: \(tf(t,d) = c(t,d)\)

• Idea: a term is more discriminative if it occurs a lot but only in fewer documents

Let \(n_{d,t}\) denote the number of times the \(t\)-th term appears in the \(d\)-th document.

\[TF_{d,t} = \frac{n_{d,t}}{\sum_i{n_{d,i}}}\] Let \(N\) denote the number of documents annd \(N_t\) denote the number of documents containing the \(t\)-th term.

\[IDF_t = log(\frac{N}{N_t})\] TFiDF weight:

\[w_{d,t} = TF_{d,t} \cdot IDF_t\]

• How about total term frequency?
  • \(ttf(t) = \sum_d{c(t,d)}\)

• How about total term frequency?
  • \(ttf(t) = \sum_d{c(t,d)}\)

• Cannot recognize words frequently occurring in a subset of documents

• Euclidean distance

  \(dist(d_i, d_j) = \sqrt{\sum_{t\in V}{[tf(t,d_i)idf(t) - tf(t, d_j)idf(t)]^2}}\)

  • Longer documents will be penalized by the extra words

  • We care more about how these two vectors are overlapped

• Cosine similarity

  • Angle between two vectors:

    \(cosine(d_i, d_j) = \frac{V_{d_i}^TV_{d_j}}{|V_{d_i}|_2 \times |V_{d_j}|_2}\) ← TF-IDF vector

  • Documents are normalized by length

• Determine text mapping to proper names

• Determine text mapping to proper names

• Annotate each word in a sentence with a part-of-speech.

• Useful for subsequent syntactic parsing and word sense disambiguation.

• CogComp: https://cogcomp.seas.upenn.edu/vsa8080/curator

• CCG Demos: https://cogcomp.seas.upenn.edu/page/demos/

• Stanford parser: http://nlp.stanford.edu:8080/parser/index.jsp

• What is your experience level with Python?

  • I don’t know anything about Python.
  • I know a bit about Python and/or I have worked with Python years ago.
  • I am familiar with Python, but I use another programming language for my work.
  • I use Python for my daily work, but I don’t know it very well.
  • I use Python for my daily work and I know it very well, but I am not an expert.

  • I am an expert with Python.

  • Go to www.menti.com and use the code 7338 2184

• Which Python IDE do you mostly use? If you use more than one environment fill in the other text boxes.

  • Go to www.menti.com and use the code 7338 2184

• From 0 (none) to 5 (expert), how familiar are you with Google Colab?

  • Go to www.menti.com and use the code 7338 2184

• Latest: Python 3.9.1

• Follow the tutorial on Python in Google Colab for the Applied Text Mining course: link

• Python For Beginners
  • https://www.python.org/about/gettingstarted/
• The Python Language Reference
  • https://docs.python.org/3/reference/
• Python 3.9.1 documentation
  • https://docs.python.org/3/

• Colaboratory, or “Colab” for short, allows you to write and execute Python in your browser, with
  • Zero configuration required
  • Free access to GPUs
  • Easy sharing

• [Intro](https://colab.research.google.com/notebooks/intro.ipynb)

• Cheat-sheet for Google Colab

• Keyboard shortcuts:

• Text data is everywhere!
• Language is hard!
• The basic problem of text mining is that text is not a neat data set
• Solution: text pre-processing & VSM

In a few moments:

• You will be automatically added to a practical session.
• There will be a practical instructor present.
• At the end of the practical, you will be automatically returned to the main meeting.